Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study.

PURPOSE: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into six elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. METHODS: Messick's unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from three pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018-2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). RESULTS: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone, r = 0.34, P = .019. CONCLUSION: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Giving Literal Thanks: An ACGME-Sponsored Initiative to Bring Residents Back to the Bedside.

Background Resident burnout is at an all-time high. In response, the Accreditation Council for Graduate Medical Education (ACGME) developed the Back to Bedside grant for resident-led burnout interventions that increase the time residents spend with patients. Objective We designed a resident-patient reading intervention, Giving Literal Thanks (GLT), intended to increase meaningful time residents spend with patients and thereby decrease burnout. Methods All 65 pediatric residents rotating through our academic hospital's inpatient units from Fall 2019 through Fall 2021 were invited to read and gift books to their patients. We studied our intervention's relationship to resident burnout using a convergent mixed-methods design, including anonymous, unlinked pre-, peri-, and post-intervention surveys and focus groups. Qualitative and quantitative data were analyzed separately, then integrated to describe burnout pre- and post-intervention. Results Forty-one of 65 residents (63.1%) completed pre-intervention surveys, and 8 of 65 (12.3%) completed post-intervention surveys. Twenty-seven resident-patient reading interactions were recorded, and 2 focus groups were held (1 pre- and 1 post-intervention). Five themes were identified: (1) limited opportunities exist to spend time at the bedside; (2) spending time at the bedside is valuable; (3) other responsibilities may preclude time at the bedside; (4) GLT could promote positive outcomes; and (5) GLT might not be the right tool to reduce burnout. Further quantitative data analysis was prevented by low survey response rates. While GLT was positively received and feasible, we were unable to show an improvement in burnout. Conclusions GLT was well-regarded but may not improve resident burnout.

Needs Assessment of Residents and Nurses for Interprofessional Cross-Cultural Curriculum.

BACKGROUND: Clinicians must learn to care for patients from different cultures. They must also work in teams for optimal outcomes. Few studies have analysed the intersection of cross-cultural care and interprofessional education. We completed a needs assessment to determine what paediatric residents and nurses view as essential cross-cultural components. METHODS: We led a sequential mixed-methods study of residents and nurses at an urban tertiary care centre. We administered the 'Residency Training in Cross Cultural Care' instrument. Survey results informed focus group interview guide development. Thematic analysis of qualitative data was conducted following the five stages to qualitative research framework. FINDINGS: Thirty-five residents (70%) and 46 nurses (66%) completed the survey. Five residents and six nurses attended a focus group. Five themes resulted: 1) lack of clinician diversity and cross-cultural training, 2) effective cross-cultural care not always delivered, 3) multiple challenges for delivery of effective cross-cultural care, 4) call to action for enhanced cross-cultural education and 5) different cross-cultural educational needs between nurses and residents. DISCUSSION: Residents and nurses conveyed that meaningful cross-cultural care education is imperative, lacking, and must offer skills to improve patient care. They call for a robust cross-cultural curriculum that is up to date and teaches about intersectionality. CONCLUSION: We have used this data to implement interprofessional interventions to meet their unique learning needs and enhance upstander skills by using facilitated discussions of techniques, case studies and role plays. Ultimately, teaching all clinicians to care for patients of other cultures may lead to more equitable care.

Engaging Health Professions Educators in an Initiative to Create and Disseminate Micro Virtual Asynchronous Educator Development Content During COVID-19.

OBJECTIVES: Most teaching activities pivoted to virtual during COVID-19. Demanding schedules necessitated brief, asynchronous, and widely disseminated professional development to encourage connection and reinforce educator identity. We created and disseminated micro virtual asynchronous educator development while engaging educators from interprofessional backgrounds to co-create teaching tips. We undertook educational evaluation to understand the impact of our educational initiative, measured along 2 professional development dimensions offering contexts for learning-the individual or independent experience, and the group or collective experience-which served as an organizing framework from which to interpret our results. METHODS: Between June 2020 and July 2021, 74 educators at our institution were asked to participate in our "One Minute Teaching Tips" initiative by providing a brief tip or pearl. Each tip was edited, prepared as a high-quality infographic, and shared via: (1) website; (2) digital monitors; (3) newsletter; and (4) Twitter (now X). Quantitative metrics measuring website and Twitter engagement were analyzed descriptively and to determine distinctions among key variables. Participants answered a brief survey and directed content analysis was utilized to analyze the open-ended responses. RESULTS: Fifty educators (67.5%) participated. Among those, 45 (90%) completed the survey. Tips were accessed via website 1447 times, averaging 28.9/week. The average tweet garnered 43.2 engagements. Six categories aligned with the individual dimension: participants engaged meaningfully, considered prior experience, relied on educational principles, focused on interest or passion, reflected on teaching practices, and experienced reinforced professional identity. Six categories aligned with the collective dimension: participants felt the initiative supported institutional need, was visible to the community, encouraged engagement with other tips, supported inclusive participation, brought value within the community, and showcased education. CONCLUSION: Our initiative aligns with a contemporary understanding of professional development and was impactful along individual and collective dimensions. Similar initiatives could be developed in alignment with accreditation requirements.

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review.

PURPOSE: This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies. METHOD: In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency. RESULTS: The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment. CONCLUSIONS: This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

A Nurse/Intern Partnership Program Accelerates Positive Relationship Formation Between Pediatric Interns and Pediatric Nurses.

BACKGROUND: The relationship between nurses and interns affects the quality of patient care; efforts to improve this relationship are necessary for optimal care. Previously, relationship formation was found to depend on mutual trust, respect, effective communication, and undergoing the process of role formation, all of which require time. This led to the hypothesis that instituting a Nurse/Intern Partnership Program (NIPP) would accelerate relationship formation. METHODS: Using a semi-structured interview guide, prior to NIPP implementation, one nurse (n = 5) and 2 second-year resident (total n = 6) focus groups were held. After implementation, one nurse (n = 6) and one intern (n = 9) focus group were held. The focus group data were analyzed using the 5 stages of qualitative research framework. RESULTS: Thematic data analysis with 3 independent coders supported the previously developed model of relationship formation; the NIPP accelerated relationship formation between nurses and interns through creating an early ally, improving communication, and increasing role understanding. The program may have improved the culture of the pediatric unit. The interns reported more of a positive impact of the NIPP on nurse/intern relationship than the nurses. CONCLUSIONS: The NIPP was positively received by pediatric residents and nurses and successfully accelerated relationship formation between the 2 groups.

Confidence of Faculty in Writing Letters of Recommendation for Pediatric Fellowship Applicants.

BACKGROUND: The assessment of pediatric residents applying to subspecialty fellowship programs relies on faculty letters of recommendation (LOR). However, it is unclear if pediatric faculty are confident that their LOR are effective. OBJECTIVE: This study aims to assess the confidence of pediatric faculty in writing an effective LOR for pediatric residents applying to subspecialty fellowship programs. METHODS: Survey development was conducted using evidence-based best practices. Surveys were distributed via email in 2021 to all full-time pediatric faculty members who taught pediatric residents in a large academic medical center. Categorical values were compared by chi-square test. RESULTS: Eighty-five out of 150 (57%) faculty members completed the survey. Forty-one percent of participants were very confident that their LOR provided adequate content to assess residents during the application process. Confidence was associated with higher academic rank (p=0.02), frequent contact with residents (p=0.01), and writing >2 LOR in the last five years (p=0.0002). Confident LOR writers were more likely to describe their own background, details about the resident's scholarly activity, and the resident's ability to work as part of a team. Thirty-five percent of respondents reported never considering gender bias when writing LOR, whereas 28% reported always considering gender bias. Eighty-seven percent of respondents reported an interest in receiving LOR writing guidelines. CONCLUSION: Half of the faculty respondents were not very confident in their ability to write an effective LOR for pediatric residents applying for a fellowship. Faculty development and standardized instructions on writing effective LOR may be helpful both at the institutional and national levels, including the importance of considering gender bias when writing LOR.

Medical Students' Intention to Change After Receiving Formative Feedback: Employing Social Cognitive Theories of Behavior.

Introduction: We applied Azjen's theory of planned behavior (TPB) and Triandis' theory of interpersonal behavior (TIB) to understand medical students' intention to change behavior based on feedback received during an obstetrics and gynecology clerkship. Both models presume that behavioral intention is strongly related to actual behavior. Materials and Methods: We collected free-text responses from students during a year-long Feedback Focused initiative on the obstetrics and gynecology clerkship at Harvard Medical School. Students reported feedback daily and what they would change based on that feedback. We applied TPB and TIB to identify students' motivation to change. We analyzed data using directed content analysis. Results: We reviewed 1,443 feedback entries from 122 students between July 2, 2018, and May 31, 2019. Self-efficacy was the most commonly represented component, related to a student expressing their own role, ability, or skill integrating the feedback (85%). Some entries (11%) focused on students' attitudes or beliefs about the outcome of the implemented feedback, usually patient focused but sometimes about the learner's outcome. Intentions motivated by social norms and expectations focused on the perceived or stated expectations of others, usually a superior or a team (11%). A small number of entries (1.7%) indicated that students had an emotional response to challenging or meaningful feedback. Conclusions: While self-efficacy is an important change motivator, faculty development geared toward improving the provision of meaningful feedback that bridges a desired behavior change to an outcome of interest, framed through the attitudes and beliefs or social norms lens, may improve trainee performance.

Psychological safety between pediatric residents and nurses and the impact of an interdisciplinary simulation curriculum.

BACKGROUND: Effective teamwork in interdisciplinary healthcare teams is necessary for patient safety. Psychological safety is a key component of effective teamwork. The baseline psychological safety on pediatric inpatient healthcare teams is unknown. The purpose of this study is to determine the baseline psychological safety between pediatric nurses and residents and examine the impact of an interdisciplinary nighttime simulation curriculum. METHODS: A convergent, multistage mixed methods approach was used. An interprofessional simulation curriculum was implemented fall 2020 to spring 2021. Qualitative focus group data and quantitative survey data on team psychological safety were collected and compared, both pre- and post-intervention and across nurses and residents. Thematic analysis of the qualitative data was conducted, and themes integrated with survey findings. RESULTS: Data were collected from 30 nurses and 37 residents pre-intervention and 32 and 38 post-intervention, respectively. Residents and nurses negatively rated psychological safety (pre-intervention mean = 3.40 [SD = 0.72]; post-intervention mean = 3.35 [SD = 0.81]). At both times psychological safety was rated significantly lower for residents (pre-intervention mean = 3.11 [SD = 0.76], post-intervention mean = 2.98 [SD = 0.84]) than nurses (pre-intervention mean = 3.76 [SD = 0.45], post-intervention mean = 3.79 [SD = 0.50]), all P < .001. Qualitative analysis identified six integrated themes: (1) influence of existing relationships on future interactions, (2) unsatisfactory manner and frequency of communication, (3) unsatisfactory resolution of disagreements (4) overwhelming resident workload impairs collaboration, (5) interpersonal disrespect disrupts teamwork, and (6) interprofessional simulation was useful but not sufficient for culture improvement. CONCLUSION: Resident-nurse team psychological safety ratings were not positive. While interprofessional simulation curriculum shows promise, additional efforts are needed to improve psychological safety among residents and nurses.

Medical residents and attending physicians' perceptions of feedback and teaching in the United States: a qualitative study.

PURPOSE: Residents and attendings agree on the importance of feedback to resident education. However, while faculty report providing frequent feedback, residents often do not perceive receiving it, particularly in the context of teaching. Given the nuanced differences between feedback and teaching, we aimed to explore resident and attending perceptions of feedback and teaching in the clinical setting. METHODS: We conducted a qualitative study of internal medicine residents and attendings from December 2018 through March 2019 at the Massachusetts General Hospital to investigate perceptions of feedback in the inpatient clinical setting. Residents and faculty were recruited to participate in focus groups. Data were analyzed using thematic analysis to explore perspectives and barriers to feedback provision and identification. RESULTS: Five focus groups included 33 total participants in 3 attending (n=20) and 2 resident (n=13) groups. Thematic analysis of focus group transcripts identified 7 themes which organized into 3 thematic categories: (1) disentangling feedback and teaching, (2) delivering high-quality feedback, and (3) experiencing feedback in the group setting. Residents and attendings highlighted important themes in discriminating feedback from teaching. They indicated that while feedback is reactive in response to an action or behavior, teaching is proactive and oriented toward future endeavors. CONCLUSION: Confusion between the critical concepts of teaching and feedback may be minimized by allowing them to each have their intended impact, either in response to prior events or aimed toward those yet to take place.

Educational adaptation to clinical training during the COVID-19 pandemic: a process analysis.

BACKGROUND: The COVID-19 pandemic is unprecedented in terms of the extent and rapidity of the disruption forced upon formal clinical education, most notably the extensive transition of clinical skills learning to interactive video-based clinical education. METHODS: In a phenomenologic study, we used thematic analysis to explore the COVID-19 disruption to clinical training and understand processes relating to adaptation in a large academic medical center. We conducted semi-structured interviews with 14 clinical teachers and 16 trainees representing all levels of clinical learning. Interviews occurred within the initial three months of the crisis, and data were analyzed following a thematic analysis coding process. RESULTS: We constructed eight themes synthesizing our participants' perceptions of the immediate unanticipated disruption, noting in the process their alignment with a change management framework. These included: urgency in adapting, with an obvious imperative for change; overcoming inconsistent involvement and support through the formation of self-organized frontline coalitions; attempts to develop strategy and vision via initially reactive but eventually consistent communication; empowering a volunteer army through co-creation and a flattened hierarchy; and efforts to sustain improvement and positive momentum with celebration of trial, error, and growth. The majority of participants found positive outcomes resulting from the tumultuous change process. Moreover, they were now more readily accepting of change, and tolerant of the ambiguous and iterative nature inherent in the education change process. Many anticipated that some innovation would, or would at least deserve to, continue post- crisis. CONCLUSIONS: The COVID-19 pandemic afforded an opportunity to study the content and process of change during an active crisis. In this case of clinical education, our findings provide insight into the ways an academic medical system adapts to unanticipated circumstances. We found alignment with broader organizational change management models and that, compared with crisis management models (and their shorter term focus on resolving such crises), stakeholders self-organized in a reliable manner that carries the potential advantage of preserving such beneficial change.

Global Med Ed Café: A safe space for conversations.

BACKGROUND: Communities of practice foster knowledge sharing, scholarly collaboration, mentoring and professional identify formation. Easier access to technology overall and the desire to engage with peers during the pandemic have increased opportunities to engage in virtual global conversations. APPROACH: The Brigham Education Institute designed and implemented a series of six bimonthly, 1-hour 'Global Medical Education Cafés' between September 2020 and June 2021. They were offered on the Zoom platform and featured a core educational topic led by one or two discussants, followed by small group interactions surrounding specific questions or challenges. Discussants were selected based on their international reputation and recognition as health professions education scholars and leaders with expertise in specific areas. Each café was advertised through newsletters, social media, educational email lists and open to educators from any healthcare discipline or geographical region. EVALUATION: Between 26 and 41 participants attended each session from 12 countries representing five continents. Content analysis of preliminary education evaluation data from chats and reflective narratives identified four key themes: (1) exciting interactive format; (2) sense of worldwide community; (3) safe space for conversations; and (4) strong moderators facilitating engaging peer conversations. IMPLICATIONS: The series leaders aimed to emphasise inclusive, non-hierarchical and cross-cultural conversations. Preliminary evaluation suggests that this goal was achieved; these will remain priorities in future series. Educators worldwide can organise similar events with minimal resources. A safe space for conversations and open-mindedness to learning from a variety of learning and societal cultures are likely essential elements for success.

The Acceptability of Avatar Patients for Teaching and Assessing Pediatric Residents in Communicating Medical Ambiguity.

Background: Simulation offers a means to assess resident competence in communication, but pediatric standardized patient simulation has limitations. A novel educational technology, avatar patients (APs), holds promise, but its acceptability to residents, educational relevance, and perception of realism have not been determined. Objective: To determine if APs are acceptable, provide a relevant educational experience, and are realistic for teaching and assessment of a complex communication topic. Methods: Pediatric residents at one academic institution participated in an AP experience from 2019 to 2021 consisting of 2 scenarios representing issues of medical ambiguity. After the experience, residents completed a survey on the emotional relevance, realism, and acceptability of the technology for assessment of their communication competence. Results: AP actor training required approximately 3 hours. Software and training was provided free of charge. Actors were paid $30/hour; the total estimated curricular cost is $50,000. Sixty-five of 89 (73%) pediatric residents participated in the AP experience; 61 (93.8%) completed the survey. Forty-eight (78.7%) were emotionally invested in the scenarios. The most cited emotions evoked were anxiety, uncertainty, concern, and empathy. The conversations were rated by 49 (80.3%) as realistic. APs were rated as beneficial for learning to communicate about medical ambiguity by 40 (65.5%), and 41 (66.7%) felt comfortable having APs used to assess their competence in this area. Conclusions: Pediatric residents were emotionally invested in the AP experience and found it to be realistic. The experience was rated as beneficial for learning and acceptable to be used for assessment of how to communicate medical ambiguity.

Fostering educator identity through social media.

BACKGROUND: Communities of practice (CoPs) are integral to professional identify formation (PIF) of educators. However, it can be challenging to identify and engage with such communities. Social media (SoMe) can facilitate formation of virtual CoPs, support PIF and increase visibility and recognition among peers. Yet, privacy concerns, lack of skills, time constraints and value doubts are reported barriers. The purpose of this study was to explore perceived benefits, challenges and suggested strategies to promote SoMe engagement from a group of health professions educators (HPEs). METHODS: Using qualitative methodology, we explored opinions of educators participating in Harvard Macy Institute courses regarding professional SoMe use through focus group discussions. Discussions were audiotaped and transcribed, and thematic analysis was performed. Brief questionnaires were distributed to collect frequency of and confidence in SoMe use; these data were analysed descriptively. The Partners Healthcare Institutional Review Board approved the study. RESULTS: Forty-eight educators participated in six focus groups. About 93.8% had at least one professional SoMe account, although engagement frequency varied. Qualitative analysis revealed three themes: (1) challenges to engagement, including juggling commitments and feeling overwhelmed; (2) benefits to joining a community, including staying informed and sharing scholarship; and (3) strategies to encourage inclusion, including tailored training and institutional champions. CONCLUSION: Though challenges to engagement were reported, participants noted multiple benefits. SoMe engagement can foster virtual CoPs among HPEs and potentially contribute to PIF. Subsequently, we offer five recommendations for efficient and impactful social media engagement for healthcare professionals.

The COVID-19 Online Learning Transition and the Plus/Delta Debriefing Model.

ABSTRACT: As a result of the COVID-19 pandemic, the Harvard Medical School Master of Medical Sciences in Medical Education program implemented the Plus/Delta debriefing model to collect real-time learner feedback and facilitate continuous quality improvement within our assessment and program evaluation module, which rapidly transitioned online. The model is grounded in quality improvement methodology and encourages bidirectional rather than top-down feedback. Learners identified specifically what went well with each module session (Plus) and how to improve for the next time (Delta). After each session, narrative responses were reviewed, aggregated into meaningful categories, and implemented when possible. Implemented improvements focused on increasing session interactivity and enhancing communication. Learners pointed to the uniqueness of the model and applauded efforts to embed them in the cocreation process. Asking learners to debrief their experience-especially during a module which is iteratively redesigned week-by-week-can offer a realistic view of what is working and what needs further refinement. As we continue to navigate the COVID-19 pandemic, rapid, continuous educational quality improvement deserves renewed emphasis. Implementing the Plus/Delta debriefing model into longitudinal online teaching sessions is a translatable, easily replicable, and free method of receiving immediate and valuable learner feedback.

The "Medical Education Roadshow:" Delivering Faculty Development to Busy Clinician Educators When They Least Expect It.

INTRODUCTION: Designing impactful faculty development for busy clinicians is challenging. Many may not recognize their impact on the learning environment or prioritize their development as educators. Our objective was to evaluate the feasibility and acceptability of a faculty development approach, the "Medical Education Roadshow," which delivered succinct, actionable faculty development at regularly scheduled, departmental clinical business meetings. METHODS: Between October 2018 and October 2019, we conducted six 15-minute "roadshows" for the Obstetrics and Gynecology faculty at one academic medical center. Each roadshow addressed a foundational education topic in an interactive manner with an emphasis on one take-away skill in teaching behavior. We utilized a simple, anonymous evaluation tool to obtain participant feedback and analyzed quantitative data descriptively and qualitative data thematically. RESULTS: A total of 174 of 265 evaluations were returned (65.6% response rate). Participants indicated that the roadshows helped them think about teaching more effectively and offered one or more practical daily practice tips. Qualitative findings coalesced into two themes. First, participants identified multiple intended practice changes, including using more effective teaching strategies, being more deliberate about feedback, and modeling exemplary professional behavior. Second, participants recommended multiple improvement opportunities and future topics. DISCUSSION: Busy clinical faculty were highly receptive to opportunities to improve as educators through the "roadshow" approach.